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Explicit  vs.  Implicit  Networks 


Typically,  we  see  edges  and  reason  about  the 

latent  properties  of  the  vertices. 


►V: :  >  Explicit  vs.  Implicit  Networks 


▼  ■•V*.** 


What  if  we  don't  observe  edges,  but  only 
noisy  emissions  from  each  vertex? 
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Functional  Connectivity  of  Neurons 
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Overview 


►  Mutually-Exciting  Point  Processes 

►  Aldous-Hoover  Graph  Priors 

►  MCMC  Inference  with  Data  Augmentation 

►  Application  Examples 

►  Extending  for  Neural  Models  with  Inhibition 
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Multivariate  Point  Processes 


►  The  point  process  is  a  foundational  statistical  object. 

►  Gives  us  random  subsets  of  a  larger  space. 

►  Many  data  are  well  modeled  as  point  processes: 

►  Seismology 

►  Epidemiology 

►  Economics 

►  Modeling  dependence  is  challenging  -  "beyond  Poisson" 

►  Strauss  and  Gibbs  Processes 

►  Determinantal  and  Permanental  Point  Processes 

►  TODAY:  Mutually-Exciting  (Hawkes)  Processes 
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Point  Process 


►  A  point  process  on  X  gives  us  random  subsets  {*n}„=1. 


►  Formally,  a  random  locally-finite  counting  measure. 

►  Most  of  the  time  we  think  of  them  as  giving  us  finite 
subsets  of  compact  subset  of  Rd,  e.g.,  time  or  space. 
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Poisson  Process 


►  The  Poisson  process  is  the  most  basic  point  process. 

►  Disjoint  regions  are  independent. 

►  The  number  of  points  in  a  region  is  determined  by 
integrating  the  rate  function  over  that  region. 
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Dependence  in  Multivariate  Point  Processes 


►  In  the  Poisson  process,  everything  is  conditionally 
independent  given  the  rate  function. 


►  How  can  we  get  spike-driven  dynamics? 
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Hawkes  Process  Dynamics 
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Hawkes  Process  Dynamics 
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Hawkes  Process 


►  The  Hawkes  process  specifies  conditional  Poisson 
dynamics  in  causal  cascades. 

►  Hawkes  -  Biometrika  (1971),  J.  RSS-B  (1971) 

►  Hawkes  -  Stochastic  Point  Processes  (1972) 

►  Hawkes  &  Oakes  -  J.  Applied  Prob.  (1974) 

►  Purely  excitatory:  each  spike  increases  the  intensity 
according  to  a  weighted  temporal  kernel. 

►  Self-excitation:  increase  your  own  rate. 

►  Mutual-excitation:  increase  other  rates. 

►  Spectral  conditions  ensure  stability. 
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Graph  Structure  from  Hawkes  Dynamics 
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Graph  Structure  from  Hawkes  Dynamics 
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Graph  Structure  from  Hawkes  Dynamics 
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Bayesian  Inference 


►  The  Hawkes  process  provides  a  likelihood  to  connect 
hypotheses  about  an  unobserved  graph  to  observed 
event  data. 


►  With  a  Bayesian  model,  we  can  manipulate  the  posterior 
distribution  over  graphs,  marginalizing  out  nuisance 
parameters. 

►  We  can  infer  the  temporal  kernels  that  modulate  the 
interaction. 


►  We  can  perform  model  comparison  between  different 
random  graph  models  and  their  properties. 
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Exchangeable  Random  Graphs 


►  Recent  work  on  exchangeable  random  graphs  provides 
a  rich  set  of  priors  for  underlying  networks,  e.g., 
Diaconis  &  Janson  (2007),  Orbanz  &  Roy  (2013). 


►  Aldous-Hoover  unifies  many  existing  graph  models: 


20  40  60 

Postsynaptic  Neuron 

Erdos-Renyi 


.  T.T- 

•  -  \  • 

:  •  .  " 
■  ■  ■  ■ 

■  •  8 

s  1*  * 

•  a 

• 

• 

1  • 

I- 

\ 

:#:s 

>  V-* 

• 

•  ■ 

•  •  f 

•  1  m 

.'A.  J 
*  * 

• 

■ 

•  B  B  4 

m  a 

1 

■  •  »  1 

20  40  60 

Postsynaptic  Neuron 


Stochastic  Block 
Model 


20  40  60 

Postsynaptic  Neuron 


Latent  Distance 
Model 


Saturday,  August  3,  13 


4  w  ^ 

44«M4 


* 
ft 


4  4 


4 


Hawkes  Process  Formalism 


►  K nodes  with  events  in  [0,  T\. 

►  N  ordered  event  times  sn  e  [0,  T] . 

►  Node  of  event  n  given  by  cn  e  {1, 2,  •  •  •  ,  A}. 

►  Base  rates  A?  (t) 

POO 

►  Kernel  ge(t),  such  that  <  0)  =  0  and  /  ge(t)  dt 

Jo 


►  Binary  adjacency  matrix  A  e  {0, 1/  xK 

►  Interaction  weight  matrix  W  e  x  K 


►  An  event  on  k  induces  Wk,k'Ak,k'  expected  events  on  k! . 


Saturday,  August  3,  13 


^  4.  A.  “ 


A  4 

A 
*4 


*  W 


lr  •% 


AS 


A 


Hawkes  Process  with  Data  Augmentation 


►  Instantaneous  Poisson  rate: 


N 

A k(t)  A °(t)  +  ^I(sn  <  t)ACnjkWCnjkge(t  sn) 

n= 1 


►  The  superposition  property  of  Poisson  processes  means 
that  each  event  is  explained  by  either  the  background 
rate  or  exactly  one  previous  event. 

►  Use  Z  e  { 0,l}iVxiVto  represent  these  latent  variables, 
where  Znn/  =  1  means  that  event  n  induced  event  n'. 
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Data- Augmented  Likelihood 


KK,  Z\A,W,  {\m)K=1,0)  = 


0 


K 


Poisson  process  likelihood  for  events 
from  the  background  process. 
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Data- Augmented  Likelihood 


P({Sn,C„} 

K 

II  exp 

k= 1 


Poisson  process  likelihood  for  events 
induced  by  previous  events. 
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Data- Augmented  Likelihood 


Ugly  looking,  but  just  normalization  constants. 
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Modeling  Details 


►  Logistic-normal  impulse  response 

►  Reasonably  flexible,  with  compact  support. 

►  Gaussian  process  for  log  background  rate 

►  Smoothly-varying  or  periodic  external  effects. 

►  Conjugate  gamma  priors  for  weights 

►  Can  also  be  coupled  via,  e.g.,  latent  block  identity. 
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Inference  with  MCMC 


►  Graph  structure:  collapsed  block  Gibbs 

►  Edge  weights:  Gibbs  (conjugate  gamma  posterior) 

►  Latent  parent  explanations:  parallel  Gibbs 

►  Background  rates:  elliptical  slice  sampling 

►  Temporal  kernels:  Gibbs  or  slice  sampling 


►  Many  transitions  can  be  efficiently  simulated  on  a 
GPU,  enabling  models  with  hundreds  of  nodes  and 
millions  of  events. 
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Financial  Upticks  /  Downticks 


10  stocks,  finance  and  tech  sectors,  intraday 
price  moves  over  one  week  in  Sept  2009. 


Interaction  between  Finance  and  Technology  Stocks 
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Inferred  Block  Structure  (SBM+GP) 
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Homicides  and  Armed  Batteries  in  Chicago 
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Murder  and  Battery  in  Chicago  (2001-2012) 
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Gang  territories  identified  by  CPD 


42.05 


41.95 


41.9  - 


41.6 


W+nnette 


Efenstjon 


Mir': 


lir  f:?! 


V  v: 


Linrnlnfcvnnrl 


* . 


|  *  :  VSn 


S3 


v 


EUrwmod 


HrHkvriiiH  h 


■ 


Chicago 


etxrtrfcirl 

Lyrns 


LaG^q  = 


v 


Hnr1g.-.ir-A 

;  Oi  irlj.-nk, 

•  =-  ‘_ 


Hilln 


TBE 


reeo 


SS 


r 


Dolton 


i 


Em  am  L 


41 .85 

h!  ... 

'Lt  r .  J"i  q  c 


41.8 


41.75  - 


41.7 


41.65 


-87.9  -87.85  -87.8  -87.75  -87.7  -87.65  -87.6  -87.55  -87.5 

Longitude 


Gangster  Disciples 
|  Black  Disciples 
Black  P  Stone  &  Mickey  Cobras 
Mafia  Insane  Vice  Lords 
Black  P  Stone 
Four  Corner  Hustlers 
Ambrose 
Latin  Kings 

Conservative  Vice  Lords 

Black  Disciples,  Black  P  Stone,  Conservative  Vice  Lords 
]  Four  Corner  Hustlers,  Black  P  Stone,  and  CBL 
]  Gangster  Disciples  and  Black  Disciples 
]  New  Breed 
Maniac  Latin  Disciples 
Party  People 
Satan  Disciples 
Two-Six 
Latin  Saints 
Black  Souls 
Two-Two  Boys 
Traveling  Vice  Lords 
Unknown  Vice  Lords 
Spanish  Cobras 
Latin  Jivers 
Orchestra  Albany 
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Alliances:  Folk  Nation  vs.  People  Nation 
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Gang  Affiliations  as  Identified  by  Chicago  PD 
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Seasonal  Variation 


Homicide  rates  in  Chicago  gang  territories  (2001-9/2012) 
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Inferred  Interacting  Territories 
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Inferred  Gang  Territories  and  Interactions 
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We  also  infer  the  event 
sources,  using  a  spatial 
model  and  a  Dirichlet 
process  prior. 
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Trying  to  Predict  Hotspots 
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Predicted  Homicide  Rate,  Memorial  Day  2012 
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Neural  Modeling  with  Inhibition 


►  Neural  functional  connectivity  involves  both  excitation 
and  inhibition,  so  the  pure  Hawkes  is  inappropriate. 

►  We  extend  the  model  to  allow  for  negative  weights  and 
include  a  saturating  nonlinearity. 

►  This  effective  becomes  a  Bayesian  variant  of  the 
popular  generalized  linear  model  (GLM)  from  the 
computational  neuroscience  literature. 

►  We  can  leverage  graph  priors  within  this  framework  to 
discover  latent  neural  properties  and  connectivity. 
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Bayesian  GLM  for  Neural  Data 


"Background" 


Saturating  Nonlinearities  Poisson  Processes 


(Scalar) 
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Retinal  Ganglion  Cells 


27  neurons,  50K  spikes,  macaque  retina 
(Data  from  Pillow  and  Chichilnisky) 
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Inferred  Network  Properties 


RGC  Connectivity 


Postsynaptic  Neuron 


Outgoing  Weights  (OFF) 


Pr(A)  vs  Distance 
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Summary 


►  We  cannot  directly  observed  edges  for 
many  networks  of  interest. 

►  However,  these  latent  graphs  can  be 
inferred  from  vertex  emissions. 

►  The  purely-excitatory  case  (Hawkes 
process)  enables  an  elegant  data- 
augmentation  approach  to  inference. 

►  MCMC  is  fast  and  tractable,  and  lets  us 
reason  about  different  graphs  and  graph 
priors. 

►  The  inhibitory  case  is  less  tractable,  but 
important  for  neuroscience  applications. 


Scott  Linderman 
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